Finding  a  Needle  in  a 
PCAP 

Flocon  2015 


Emily  Sarneso 


Carnegie  Mellon 


©  2013  Carnegie  Mellon  University 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1 .  REPORT  DATE  2.  REPORT  TYPE 

27  JAN  2015  N/A 

3.  DATES  COVERED 

4.  TITLE  AND  SUBTITLE 

5a.  CONTRACT  NUMBER 

Finding  a  Needle  in  a  PCAP 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

Sarneso  /Emily 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Software  Engineering  Institute  Carnegie  Mellon  University  Pittsburgh, 

PA  15213 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release,  distribution  unlimited. 

13.  SUPPLEMENTARY  NOTES 

The  original  document  contains  color  images. 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

'VPSITT?  act 

1 8 .  NUMBER  1 9a.  NAME  OF 

DACPC  T?Th  QPrAMQTPT  U  DUDCHM 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  S  AR 

unclassified  unclassified  unclassified 

27 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Copyright  2014  Carnegie  Mellon  University. 


This  material  is  based  upon  work  supported  by  the  Department  of  Defense  under  Contract  No.  FA8721-05-C-0003  with  Carnegie  Mellon 
University  for  the  operation  of  the  Software  Engineering  Institute,  a  federally  funded  research  and  development  center. 


Any  opinions,  findings  and  conclusions  or  recommendations  expressed  in  this  material  are  those  of  the  author(s)  and  do  not  necessarily 
reflect  the  views  of  the  United  States  Department  of  Defense. 


References  herein  to  any  specific  commercial  product,  process,  or  service  by  trade  name,  trade  mark,  manufacturer,  or  otherwise,  does 
not  necessarily  constitute  or  imply  its  endorsement,  recommendation,  or  favoring  by  Carnegie  Mellon  University  of  its  Software 
Engineering  Institute. 


NO  WARRANTY 

THIS  CARNEGIE  MELLON  UNIVERSITY  AND  SOFTWARE  ENGINEERING  INSTITUTE  MATERIAL  IS  FURNISHED  ON  AN  “AS-IS” 
BASIS.  CARNEGIE  MELLON  UNIVERSITY  MAKES  NO  WARRANTIES  OF  ANY  KIND,  EITHER  EXPRESSED  OR  IMPLIED,  AS  TO 
ANY  MATTER  INCLUDING,  BUT  NOT  LIMITED  TO,  WARRANTY  OF  FITNESS  FOR  PURPOSE  OR  MERCHANTABILITY, 
EXCLUSIVITY,  OR  RESULTS  OBTAINED  FROM  USE  OF  THE  MATERIAL.  CARNEGIE  MELLON  UNIVERSITY  DOES  NOT  MAKE 
ANY  WARRANTY  OF  ANY  KIND  WITH  RESPECT  TO  FREEDOM  FROM  PATENT,  TRADEMARK,  OR  COPYRIGHT  INFRINGEMENT. 


This  material  has  been  approved  for  public  release  and  unlimited  distribution  except  as  restricted  below. 

This  material  may  be  reproduced  in  its  entirety,  without  modification,  and  freely  distributed  in  written  or  electronic  form  without  requesting 
formal  permission.  Permission  is  required  for  any  other  use.  Requests  for  permission  should  be  directed  to  the  Software  Engineering 
Institute  at  permission@sei.cmu.edu. 

Carnegie  Mellon®,  CERT  ®  ,  CERT  Coordination  Center®  and  Flocon®  are  registered  marks  of  Carnegie  Mellon  University. 

DM-0001893 


^CEFtt 


Software  Engineering  Institute  CarnegieMellon 


2 


Goal 


Describe  a  full  packet  capture  solution  that  can 
quickly  and  efficiently  produce  requested  information. 


Show  analysis  capabilities  of  YAF,  super_mediator, 
and  SiLK. 


Demonstrate  PCAP  features  in  YAF. 
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PCAP  Challenges 

Volume  (4Gbps): 

•  1  Hour:  1 .7TB 

•  1  Day:  40.8TB 

•  1  Week:  285.6TB 

•  1  Month:  1.1  PB 

Data  Stored  on  Sensors 

•  Separate  from  analysis 
Indexing: 

•  Timestamp  Files 

•  BPF  Filters 

•  GUI  tools 

•  Splunk 


(CEIW 
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YAF  PCAP  Features 


Rolling  PCAP  dump 

•  Rotates  files  using  time  or  size. 

•  Creates  meta  file  with  flows  contained  in  each  PCAP  file. 

Index  a  PCAP  File 

•  Uses  flow  key  hash  and  start  time. 

PCAP  per  flow 

•  Creates  a  PCAP  file  for  each  flow. 

•  Use  with  BPF  filters. 


(CEIW 
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GhOst  Rat  Investigation 


Software  Engineering  Institute 
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GhOst 


Chinese  remote  access  Trojan 
Free  source  code 
Easy  to  modify 

Distinctive  Network  Signature 


Signature 

Compressed 

Uncompressed 

ZLIBHDR 

Data 

Usually  5 

Length 

Length 

0x789C 

BYTES 

4  BYTES 

4  BYTES 

2  BYTES 
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Method 


29,000  (15G)  PCAP 
samples 

Use  YAF  to  index  and 
produce  flow,  DPI 

YAF  Signatures 


(CEOT  I  * 


—  Software  Engineering  Institute 


Carnegie  Mellon 


Flow 


Enhanced 

Flow 

(DPI) 


PCAP 
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Tool  setup 


*  O  O 


src  —  ssh  —  HSx25 


-Script  Help 


|!  /bin/bash 
PCAP_D I R=/storage/pcaps/* 

f  L  owcap  — sensor-conf  =/data/sensor . conf  — dest i nat i on-d i r=/data/f  L  ow  — max-f i L  e-s i ze=20m  —  L  og-dest=/var/ L  og/f  L  owp \ 
ack. Log 

super_mediator  -c  /data/super_mediator .conf  -d 
i=8 

for  file  in  $PCAP_DIR 
do 

yaf  — in  SfiLe  — out  Local host  — ipfix  tcp  — ipf ix-port=6789  — Log=/var/Log/yaf . Log  — verbose  — app Label  — max-pay\ 
L  oad=1024  — p  L  ug i n-name=/usr/ L i b64/yaf /dpacketp  L  ug i n . L  a  — f  L  ow-stats  — no-stats  — app  L  abe  L -ru  L  es=/data/yaf  App  L  abe  L \ 
Rules. conf  — noerror  — pcap-meta-f i L e=/data/pcapmeta/$ (basename  $f i L e ) . txt  — observat i on-doma i n=$ i 

i=$  [$i+l] 

done 


-UU-: - FI  test.sh 


ALL  LI 


(She  L  L -script [bash] )- 


iCECT 
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Initial  Results 


'5)  $  rwstats  — fields=29  — xargs=destroy-f  Low/si  Lkfiles.txt  — top  — count  9 

[NPUT :  379068  Records  for  10  Bins  and  379068  Total  Records 


OUTPUT :  Top  9  Bins  by  Records 


jppli 1 

Records  1 

Records  1 

cumuLft 

01 

260316 1 

68.6726391 

68.672639 

80| 

43263 | 

11.4129921 

80.085631 

1391 

38170 | 

10.0694331 

90.155065 

1371 

20324 | 

5.3615711 

95.516636 

531 

1667B | 

4.3989471 

99.915582 

1191 

2401 

0.0633131 

99.978896 

33061 

68| 

0.0179391 

99.996834 

10801 

61 

0.0015831 

99.998417 

1941 

41 

0.0010551 

99.999472 

^CEFtt 
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YAF  Signatures 


Norman  ASA  2012  Report 
identifies  85  GhOst 
variants 


download0 1 .  norman .  no/docu  ments/ThemanyfacesofGhOstRat.  pdf 

Software  Engineering  Institute  Carnegie  Mellon 


9999  signature  AGh0st 
9998  signature  ALURK0 
9997  signature  A7hero 
9996  signature  AAdobe 
9995  signature  AB1X6Z 
9994  signature  ABEILa 
9993  signature  AByShe 
9992  signature  AFKJP3 
9991  signature  AFLYNN 
9990  signature  AFWAPR 
9989  signature  AFWKJG 
9988  signature  AGWRAT 
9987  signature  AG0LDt 
9986  signature  AHEART 
9985  signature  AHTTP3 
9984  signature  AHXWAN 
9983  signature  AHeart 
9982  signature  AIM007 
9981  signature  AITore 
9980  signature  AKQBBX 
9979  signature  AKrisR 
9978  signature  ALUCKK 
9977  signature  ALYRAT 
9976  signature  ALeveL 
9975  signature  ALover 
9974  signature  ALyyyy 
9973  signature  AMFYB 
9972  signature  AMoZhe 
9971  signature  AMyRat 
9970  signature  A0HMM 
9969  signature  APCRat 
9968  signature  AQWP0T 
9967  signature  ASpidern 
9966  signature  ATyjhu 
9965  signature  AURATU 
9964  signature  AW0LFKO 
9963  signature  AWangz 
9962  signature  AWinds 
9961  signature  AWorLd 
9960  signature  AX6RAT 
9959  signature  AXDAPR 
9958  signature  AXjjhj 
9957  signature  Aag@ft 
9956  signature  Aattac 
9955  signature  Acblst 
9954  signature  Ahttps 
9953  signature  AwhmhL 
9952  signature  Axhjyk 
9951  signature  A00000 
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Results  with  YAF  Signatures 


(CEIW 


(25)  $  rwstats  — fields=29  — xargs=destroy-f Low/si Lkfi Les.txt  — top  — count=50 

INPUT:  379068  Records  for  31  Bins  and  379068  Total  Records 
OUTPUT:  Top  50  Bins  by  Records 


appLi 1 

Records | 

^Records | 

cumuL^I 

01 

138766 | 

36.6071521 

36.6071521 

99691 

52080 | 

13.7389601 

50.3461121 

80| 

43263 | 

11.4129921 

61.7591041 

1391 

38170 | 

10.0694331 

71.8285371 

99991 

32076 | 

8.4618061 

80.2903441 

99891 

27998 | 

7.3860101 

87.6763541 

1371 

20324 | 

5.3615711 

93.0379251 

531 

16675 1 

4.3989471 

97.4368711 

99621 

26331 

0.6959171 

98.1327391 

99911 

21431 

0.5645431 

98.6973311 

99551 

9501 

0.2506151 

98.9479461 

99651 

8601 

0.2263721 

99.1748181 

99601 

7241 

0.1909951 

99.3658131 

99711 

3841 

0.1013011 

99.4671141 

99741 

3731 

0.0997181 

99.5668321 

99541 

3481 

0.0918041 

99.6586361 

99421 

3441 

0.0907491 

99.7493851 

99671 

1821 

0.0480121 

99.7973981 

99521 

1721 

0.0453741 

99.8427721 

1191 

1601 

0.0422091 

99.8849811 

99161 

1281 

0.0337671 

99.9187481 

33061 

68| 

0.0179391 

99.9366871 

99381 

64| 

0.0168841 

99.9535701 

99441 

62| 

0.0163561 

99.9699261 

99451 

60| 

0.0158281 

99.9857551 

99501 

28| 

0.0073871 

99.9931411 

99271 

12| 

0.0031661 

99.9963071 

10801 

61 

0.0015831 

99.9978901 

1941 

41 

0.0010551 

99.9989451 

99191 

21 

0.0005281 

99.9994721 

99791 

21 

0.0005281 

100.0000001 
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Super_mediator 


A  very  configurable  IPFIX  mediator 

Collects  every  IPFIX  information  element  YAF  can 
export 

Multiple  exporters 
Multiple  collectors  (v.1.0) 


YAF 

SUPER 

MEDIATOR 

SiLK 


File 

Storage 


(q^^j  _  Software  Engineering  Institute  CarnegieMellon 
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Super_mediator  configuration 


Listing  application  label 
first  allowed  for  quick 
binning  by  variant. 


Super_mediator  Results: 

•  227,833  Total  Bi-flows 

•  60,816  Bi-flows  GhOst 

•  86,053  Unidentified 


Application 

Hash 

Stimems 

Domain 

Sip 

Dip 

Sport 

Dport 

Protocol 

vlanint 

Iflags 

Uflags 

Riflags 

Ruflags 

Pkts, 

Rpkts 


Bytes 

Rbytes 

Data  bytes 

Rdata  bytes 

Small  pkts 

Rsmallpkts 

Largepkts 

Rlargepkts 

Nonemptypkts 

Rnonemptypkts 

Maxsize 

Rmaxsize 

Firsteight 
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Finding  a  Pattern 


eoo 

t  i  src  —  ssh  —  156x35 

file  Edit  Options  Buffers  Tools  Help 

9969 1 2829634012 1 1342871176989 1 82 
9969 1 2829699548 1 1342871137012 1 82 
9969 1 2829765084 1 1342871197036 1 82 
9969 1 2829330620 1 1342871207079 1 82 
9969 1 2829371363 1 1342871217103 1 82 
9969 1 2829437404 1 1342871227127 1 82 
9969 | 2829502940 | 1342871237150 | 82 
9969 1 2829568476 1 1342871247174 1 82 
9969 1 2829109724 1 1342871257246 1 82 
9969 1 2829175260 1 1342871267533 1 82 


9999 1 209172020 1 1393593169945 


9999 1 205960756 1 1393593324721 
9999 1 206026292 1 1393593334351 
9999 1 206091323 1 1393593344992 
9999 1 206157364 1 1393593355113 
9999 | 206222900 | 1393593365235 
9999 | 206288436 | 1393593375355 
9999 1 206353972 1 1393593385477 
9999 1 206419508 1 1393593395598 
9999 1 206435044 1 1393593405720 


84 


1060 

1061 

1062 

1063 

1064 

1065 

1066 

1067 

1068 
1069 
1047 


9999 | 209303092 

1 1393593193113 

1 84  U 

1 1049 1 8008 1 6 1 0 1 3 

1  APR 

1  AS 

IAPIEI 

131 

1399 

13191 

1191 

1191 

10 

101 

101 

101 

111 

11 

11911 

1191102 

9999 | 209434164 

1 1393593213385 

1 84  U 

1 1051 1 3008 1 6 1 0 1 8 

1  APR 

1  AS 

IAPIBI 

1 3 1 

1392 

13121 

1184 

1184 

10 

1  0  1 

1  0  1 

1  0  1 

111 

11 

|184| 

1 184 1 02 

9999 | 209499700 

1 1393593223507 

|84|2 

1 1052 1 3003 1 6 1 0 1 S 

I  APR 

I  AS 

1 AP | 5 | 

1 3 1 

|399 

|319| 

|191 

|191 

10 

1 0 1 

1 0 1 

1 0 1 

111 

11 

1 191 1 

1 191 1 02 

9999 | 209565236 

1 1393593233630 

|84|; 

1 1053 1 3003 1 6 1 0 1 3 

1  APR 

IAS 

IAPIEI 

131 

1399 

13191 

1191 

1191 

10 

101 

101 

101 

111 

11 

11911 

1191102 

9999 | 209630772 

1 1393593243752 

1 34  U 

1 1054 1 3003 1 6 1 0 1 3 

1  APR 

IAS 

IAPIBI 

131 

1399 

13191 

1191 

1191 

10 

101 

101 

101 

111 

11 

11911 

1191102 

9999 | 209696308 

1 1393593253871 

1 84  U 

1 1055 1 3003 1 6 1 0 1 3 

1  APR 

|A3 

IAPIEI 

131 

1399 

13191 

1191 

1191 

10 

101 

101 

101 

111 

11 

11911 

1191102 

9999 | 205567540 

1 1393593263994 

|84|2 

1 1056 1 3003 1 6 1 0 1 3 

1  APR 

|A3 

IAPI8I 

161 

1737 

I6B7I 

1409 

1409 

12 

121 

101 

101 

141 

14 

11911 

1 191 1 aa 

9999 | 205633076 

1 1393593274114 

1 34  U 

1 1057 1 8008 1 6 1 0 1 5 

1  APR 

IAS 

IAPIBI 

131 

1399 

13191 

1191 

1191 

10 

101 

101 

101 

111 

11 

11911 

1191102 

9999 | 205698612 

1 1393593284235 

1 34  U 

1 1053 1 3003 1 6 1 0 1 3 

1  APR 

IAS 

IAPIBI 

131 

1399 

13191 

1191 

1191 

10 

101 

101 

101 

111 

11 

11911 

1191102 

9999 | 205764148 

1 1393593294357 

1 34  U 

1 1059 1 3003 1 6 1 0 1 3 

I  APR 

I  AS 

|AP|B| 

1 3 1 

|399 

|319| 

|191 

|191 

10 

1 0 1 

1 0 1 

1 0 1 

111 

11 

1 191 1 

1 191 1 02 

9999 | 205829684 

1 1393593304477 

1 34  U 

1 1060 1 3003 1 6 1 0 1 3 

1  APR 

IAS 

IAPIBI 

131 

1399 

13191 

1191 

1191 

10 

101 

101 

101 

111 

11 

11911 

1191102 

9999 | 205895220 

1 1393593314599 

1 34  U 

1 1061 1 3003 1 6 1 0 1 3 

1  APR 

IAS 

IAPIEI 

131 

1399 

13191 

1191 

1191 

10 

101 

101 

101 

111 

11 

11911 

1191102 

1062 

1063 

1064 

1065 

1066 

1067 

1068 

1069 

1070 


8016 
3016 
3016 
3016 
3016 
80 1 6 
80 1 6 
8016 
3016 
3016 
3003 


3003 

3003 

3003 

3003 

8008 

8008 

8008 

3003 

3003 


APR  I  AS 
APR  I  AS 
APR  I  AS 
APR  I  AS 
APR  I  AS 
APR  I  AS 
APR | AS 
APR  I  AS 
APR  I  AS 
APIASI 
APRS 


I  API  5 
I  API  5 
I  API  5 
I  API  5 
I  API  5 
I  API  5 
I  AP|  5 
I  API  5 
I  API  5 
AP 1 4 1 
IASI 


AP 


APR 

APR 

APR 

APR 

APR 

APR 

APR 

APR 

APR 


3 1 408 1 328 1 200 1 200 1 0 1 0 1 0 
3 1 403 1 328 1 200 1 200 1 0 1 0 1 0 
3 1 403 1 328 1 200 1 200 1 0 1 0 1 0 
3 1 407 1 327 1 199 1 199 1 0 1 0 1 0 
3 1 403 1 328 1 200 1 200 1 0 1 0 1 0 
3 1 403 1 328 1 200 1 200 1 0 1 0 1 0 
3 | 407 | 327 | 199 | 199 | 0 | 0 | 0 
3 1 408 1 328 1 200 1 200 1 0 1 0 1 0 
3 1 410 1 330 1 202 1 202 1 0 1 0 1 0 
1 370 1 330 1 202 1 202 1 0 1 0 1 0 1 
1 9 1 6 1 310 1 682 1 434 1 434 1 2 1 


1 0 1 1 1 1 1 200 1 200 1 02 
1 0 1 1 1 1 1 200 1 200 1 02 
1 0 1 1 1 1 1 200 1 200 1 02 
1 0 1 1 1 1 1 199 1 199 1 02 
I  0  1 1 1 1 1 200 1 200 1  02 
I  0  1 1 1 1 1  200  I  200  I  02 
1 0 1 1 1 1 1 199 1 199 1 02 
1 0 1 1 1 1 1 200 1 200 1 02 
1 0 1 1 1 1 1 202 1 202 1 02 
0 1 1 1 1 1 202 1 202 1 02 
2 1 0 1 0 1 4 1 4 1 216 1 216 1 aa 


-UU-: - FI  9.txt 


20 1 13 1 1403 1 1323 
5 1 3 1 399 1 319 1 191 
5 1 3 1 399 1 319 1 191 
5 | 3 | 399 | 319 | 191 
5 | 3 | 399 | 319 | 191 
5 1 3 1 399 1 319 1 191 
5 1 3 1 399 1 319 1 191 
5 1 3 1 399 1 319 1 191 


14 1 12 1 1070 1 990 1 502 1 502 1 8 1 8 1 0 1 0 1 10 1 10 1 191 1 191 1 aa 


595 1 595 1 14 1 14 1 0 
1911010 
1911010 
191 | 0 | 0 
191|0|0 
1911010 
1911010 
1911010 


>101011 

>101011 

>|0|0|1 

>101011 

>101011 

>101011 

>101011 


0116116 
1911191 
1911191 
191 | 191 
191 | 191 
1911191 
1911191 
1911191 


191 1 191 1 aa 

02 

02 

02 

02 

02 

02 

02 


1%  L202  (Text)- 
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Analysis  Part  1 

Remove  unwanted  flows  from  unidentified  flows: 

•  Remove  flows  with  source/destination  port  138,139. 

•  Remove  flows  with  initialTCPFIags  =  ‘R’ 

•  Remove  flows  with  dataByteCount  =  0 
Find  flows  with  pattern: 

•  No  more  than  1  small  packet  (forward),  0  reverse 

•  Non-empty  packets  =  1  or  2  (forward),  1  reverse 

•  maxPacketSize  =  reverseMaxPacketSize 

•  firstEightPacketDirection  =  0x02 
Results: 

•  44,468  bi-flows  removed 

•  37,500  bi-flows  with  pattern 

•  4,085  bi-flows  did  not  follow  pattern 

£e>  Software  Engineering  Institute  CamegieMellnn 


16 


Finding  GhOst  Variants  and  Signatures 


Ann 

Stream  Content 


\  follow  TCP  Stream 


OOQOOOOO  10 

01  OO  O0  01  00  00 

40 

5-7  4b  47  70  1 

DC  4b  _ 8.  .  .  =V*.lGif,K 

DOOOQQiO 

■  L 

D000002C 

„ 

OOOOGQQC 

h 

QOOOOQOC- 

t 

00OOOQ5C 

ooooooet 

D00OCG7< 

QOOOOO&' 

0000009C 

OOQOOOA1 

OOQOOOBC 

ooooooco 

OOQOGODt 

ooooooe 

DQOOCGFC 

T 

DOOOOtOu  1 

qv  -“4  4J.  si  qi 

u=  1  y  UO  ft  . 

V-/  -  r  1  Ml  f-p  t  p  ‘  •  i  +  ♦  i?  f 

0OOOOOQO 

10  01  00  00  30  01  00 

GO 

46  57  4b  45  47 

70  =*:  4h  _ 8,.,  FUKJGX.K 

QOOOOD10 

S3 

.  .L 

00000020 

AT 

L. 

00000030 

a  I 

It . 

00000040- 

6t 

-  .t 

00000050 

34 

I).  , 

00000050 

07  i-'  <■_  ■_  Vv  _j»_  iv  -wsj 

/l  vL  /4 

J4  J.d  tM  .  .  .4.  .  3  •. ^r--  .  t.  .  J 

«■!. «"u "a «r% 

.  ..  «»-■  i  r-  ■■  p-p  p-  n  r-w i  .  _ 

«"k 

— n—  -«  — .  i  --J-V  I-Ai 

nun  i-r  k  .. 

Fr.tre  conversation  (S44  bytes) 


...  f  ind  ^  Save  As    P  nnt 


o  ASCII  O  EBCDIC  ©  Hm  Dump  Q  C  Arrays  Q  Raw 


#Hetp 

i - 


|V|  Fitter  out  this  stream 

3^  Dose 

■o - 

- 4 

iCECT 
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Analysis  Part  2 

Run  unidentified  PCAP  files  through  YAF  again  and 
export  first  100  bytes  of  payload 


2013- 09-03  08:44:25.38512013-09-03  08:44:25.6241  0.2391  0.0001  6 1 

1  I  76781  3|  564|00|00:00:00:00:00:00| 

->  0000:  ae  01  00  00  b4  02  00  00  46  57  4b  4a  47  48  00  00  FWKJGH. . 

->  0010:  00  00  00  78  9c  7d  52  3d  4b  @3  41  10  7d  77  e2  67  . . .x.}R=K.A.}w.g 

->  0020:  15  ae  10  44  c4  03  2d  44  44  62  b4  33  90  a4  . .  .D .  .-DDb  .3 . . 

<-  0000:  ae  01  00  00  b4  02  00  00  46  57  4b  4a  47  48  00  00  . FWKJGH. . 

<-  0010:  00  00  00  78  9c  7d  52  3d  4b  03  41  10  7d  77  e2  67  . . .x.}R=K.A.}w.g 

<-  0020:  15  ae  10  44  c4  03  2d  44  44  62  b4  33  90  a4  . .  .D.  .-DDb. 3. . 

2014- 08-26  22:16:12.99912014-08-26  22:16:26.0451  13.0461  0.0001  6 1 

I  14781  3 1  383|00|00:00:00:00:00:00| 

->  0000:  ff  00  00  00  5c  01  00  00  7a  9a  4e  _  \...z.N 


-> 

-> 


<- 

<— 


0010:  60  17  63  98  c4  c3  c2  c6  0b  cc  85  47  ae  c8  c4  c4 
0020:  c6  0d  a7  3b  57  3a  cf  3a  90  51  12  03  la  95 
0000:  ff  00  00  00  ■  5c  01  00  00  7a  9a  4e 

0010:  60  17  63  93  c4  cS  c2  c6  0b  cc  85  47  ae  cS  c4  c4 

0020:  c6  0d  a7  8b  57  8a  cf  3a  90  51  12  03  la  95 


'  .c _ 

. G.. 

. . . . 

_ z. 

'  .c _ 

....G... 

....W..I 

: -0. - 

51 


51 


API 


APRS  | 


I  1042 |  4 | 

AS |  API 5dlf 2bc8 1 alelcf a3 1 000 1 


604 | 00 | 00:00 :00 :00 :00 :00 I 
0 1 000 | 000 | eof  I  Cl 


II  10371  61 

AS |  API 99b9cc92 1 908f  cc9b 1 000 1 


511|00|00:00:00:00:00:00| 
0 | 000 | 000 | | Cl 


(cEKV 
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Results 


Identified  several  signature  variants  of  GhOst 

Found  55  new  GhOst  variants 

Created  YAF  Application  Label  for  GhOst 

•  Correctly  identifies  97%  of  GhOst  traffic. 


£e>  Software  Engineering  Institute  CamegieMdlnn 
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Searching  for  GhOst  in 
DEFCON  CTF  PCAP 


Software  Engineering  Institute 


Carnegie  Mellon 
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DEFCON  CTF  PCAP  Data 


0  o  o 


5rc  —  ssh  —  1 


Goal:  Test  new  GhOst 
application  label 

Defcon  CTF  PCAP  Data 

•  409  GB 

•  Separated  by  team  and 
day 


(18)  $  /usr/bin/rwstats  — fields=29  — xargs=si Ik/si Lk 

INPUT:  82586983  Records  for  27  Bins  and  82586983  Total  Recor 


DUTPUT: 

Top  50  Bins  by  Records 

:ipp  l  i  1 

Records  I 

Records  I 

cumul_K 

el 

478935341 

57.9916261 

57.991626 

53| 

193156351 

23.3882321 

81.379857 

88 1 

72723641 

8.8057021 

90.185560 

1431 

42060681 

5.0928951 

95.278454 

4431 

33404631 

4.0447811 

99.323236 

4271 

5045231 

0.6108991 

99.934135 

67| 

200091 

0.0242281 

99.958363 

22| 

104501 

0.0126531 

99.971016 

211 

74201 

0.0089841 

99.980000 

1371 

54231 

0.0065661 

99.986567 

50041 

37831 

0.0045811 

99.991148 

1941 

31221 

0.0037801 

99.994928 

68811 

19881 

0.0024071 

99.997335 

1391 

6421 

0.0007771 

99.998112 

1191 

3601 

0.0004361 

99.998548 

5541 

2901 

0.0003511 

99.998899 

1611 

2471 

0.0002991 

99.999198 

3891 

1531 

0.0001851 

99.999384 

52221 

1251 

0.0001511 

99.999535 

50601 

1141 

0.0001381 

99.999673 

69| 

1141 

0.0001381 

99.999811 

qg.  999860 

99971 

38| 

1 

0.0000461 

^9^9906 

.  ft  fi  QQQQC9 

25| 

28| 

0.0000341 

yy ,yyyyD£ 

99.999985 

1101 

el 

0.0000101 

99.999995 

59001 

41 

0.0000051: 

100.000000 

^CEFtt 
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Investigating  “GhOst”  in  DEFCON 


0  O  O  src  —  ssh  —  158x61 

(23)  $  /usr/bin/rwf i Iter — application=9997 — pass-dest=stdout — xargs=si Lk/si Ikf i Les.txt  I  /usr/bin/rwsi lk2ipf ix  I  /analysis/ecoff/bin/getFlowKeyHash 


SIPI 

dIP  | 

sPort | 

dPort | 

pro  | 

vlanl 

hashl 

ms 

10.5.19.1041 

10.5.18.21 

533881 

88881 

61 

01 

3498853266 1 

1407517727292 

10.5.18.21 

10.5.19.1041 

88881 

53388 | 

6| 

SI 

582529446 | 

1407517727292 

10.5.19.1111 

10.5.1.21 

526461 

88881 

61 

01 

3450228885 1 

1407526153262 

10.5.1.21 

10.5.19.1111 

88881 

52646 | 

6| 

01 

582533003 1 

1407526153262 

10.5.1.21 

10.5.18.21 

533881 

88881 

6| 

01 

3498856952 1 

1407517727292 

10.5.18.21 

10.5.1.21 

88881 

53388 | 

6| 

01 

582525900 1 

1407517727292 

10.5.1.21 

10.5.11.1111 

590141 

47989 | 

6| 

SI 

3867595096 | 

1407622961660 

10.5.11.1111 

10.5.1.21 

479891 

59014 | 

6| 

01 

3145043115 | 

1407622961660 

10.5.6.21 

10.5.11.1131 

595031 

80| 

6| 

0  | 

3899616611 1 

1407623040275 

10.5.11.1131 

10.5.6.21 

80| 

59503 | 

6| 

01 

5277020 | 

1407623040275 

10.5.6.21 

10.5.11.1171 

595411 

90011 

6| 

01 

3902098974 1 

1407623048722 

10.5.11.1171 

10.5.6.21 

9001| 

59541 | 

6| 

SI 

589923746 | 

1407623048722 

10.5.13.21 

10.5.11.1131 

595781 

4431 

61 

01 

3904530312 1 

1407623055575 

10.5.11.1131 

10.5.13.21 

4431 

59578 | 

6| 

01 

29068937 1 

1407623055575 

10.5.9.21 

10.5.11.1131 

595871 

22931 

6| 

01 

3905120966 1 

1407623058137 

10.5.11.1131 

10.5.9.21 

22931 

59587 | 

6| 

01 

150309616 | 

1407623058137 

10.5.12.21 

10.5.11.1171 

596991 

88881 

6| 

01 

3912451471 1 

1407623093677 

10.5.11.1171 

10.5.12.21 

88881 

59699 | 

6| 

0  | 

582520324 1 

1407623093677 

10.5.3.21 

10.5.11.1201 

597461 

1431 

6| 

01 

3915540661 1 

1407623114721 

10.5.11.1201 

10.5.3.21 

1431 

59746 | 

6| 

01 

9404760 | 

1407623114721 

10.5.2.21 

10.5.9.21 

574321 

88881 

6| 

0| 

3763882488 | 

1407523627665 

10.5.9.21 

10.5.2.21 

88881 

57432 | 

61 

01 

582519576 1 

1407523627665 

10.5.14.1071 

10.5.11.1111 

590141 

47989 | 

6| 

01 

3867598385 1 

1407622961660 

10.5.11.1111 

10.5.14.1071 

479891 

59014 | 

6| 

01 

3145040834 1 

1407622961660 

10.5.14.1071 

10.5.11.1131 

595031 

80| 

6| 

01 

3899614474 1 

1407623040275 

10.5.11.1131 

10.5.14.1071 

80| 

59503 | 

6| 

0| 

5279029 | 

1407623040275 

10.5.14.1071 

10.5.11.1171 

595411 

90011 

6| 

01 

3902097015 | 

1407623048722 

10.5.11.1171 

10.5.14.1071 

90011 

59541 | 

6| 

0  | 

589925835 1 

1407623048722 

10.5.14.1071 

10.5.11.1131 

595781 

4431 

6| 

01 

3904529633 1 

1407623055575 

10.5.11.1131 

10.5.14.1071 

4431 

59578 | 

6| 

01 

29068768 1 

1407623055575 

10.5.14.1071 

10.5.11.1131 

595871 

22931 

61 

01 

3905121711 | 

1407623058137 

10.5.11.1131 

10.5.14.1071 

22931 

59587 | 

61 

01 

150310297 | 

1407623058137 

10.5.14.1071 

10.5.11.1171 

596991 

88881 

6| 

01 

3912452070 1 

1407623093677 

10.5.11.1171 

10.5.14.1071 

88881 

59699 | 

6| 

01 

582519917 1 

1407623093677 

10.5.14.1071 

10.5.11.1201 

597461 

143| 

6| 

0| 

3915539932 | 

1407623114721 

10.5.11.1201 

10.5.14.1071 

1431 

59746 | 

6| 

01 

9407537 | 

1407623114721 

10.5.14.21 

10.5.1.21 

526461 

88881 

6| 

0  | 

3450228216 1 

1407526153262 

10.5.1.21 

10.5.14.21 

88881 

52646 | 

6| 

01 

582525670 1 

1407526153262 
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YafMeta2Pcap 


Input: 

•  Large  PCAP  file  or  list  of  PCAP  files 

•  PCAP  meta  file  created  by  YAF 

•  Flow  key  hash  and  start  time 

Output 

•  PCAP  file  with  desired  flow 


e  o  o 

Stream  Content 


|\|  Follow  TCP  Stream 


DmdT . 

. AMQ _ A _ ~.z _ ! 

- y. . 

.  .D/. 

. os . &Bh . . 

.  .  n .  . 

.  .Z.X. .?.  .r.O.  .  .6.| 

Entire  conversation  (99  bytes) 


Find 


0  Save  As 


JZL 


Print 


O  ASCII  O  EBCDIC  O  Hex  Dump  O  C  Arrays  ®  Raw 


**Help 


(V7  Filter  Out  This  Stream 


(q^^j  _  Software  Engineering  Institute  |  CarnegieMelkm 


Close 
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DEFCON  Analysis 


Used  YAF  signatures  to 
determine  other  flows  with 
“DmdT”  and  “eliza” 


“eliza”  was  a  text-based 
space  economy  simulator 
challenge  at  CTF 


80%  of  DmdT  traffic  went 
to  last  place  team. 


(CEOT  I  * 


—  Software  Engineering  Institute 


Carnegie  Mellon 


INPUT:  82586983  Records  for  29  Bins  and  82586983  Total  Records 

OUTPUT:  Top  30  Bins  by  Records 

agplij  Records  I  %Recordsl  cumulJSI 

. 01  401496321  48.6149641  48.6149641 

72.0031421 
81.3968001 
90.1821161 


1431 

4206068 1 

5.0928951 

95.2750111 

4431 

3340449 1 

4.0447651 

99.3197751 

4271 

504523 1 

0.6108991 

99.9306741 

671 

20009 1 

0.0242281 

99.9549021 

221 

104501 

0.0126531 

99.9675551 

211 

74201 

0.0089841 

99.9765401 

1371 

54231 

0.0065661 

99.9831061 

50041 

37831 

0.0045811 

99.9876871 

1191 

32061 

0.0038821 

99.9915691 

1941 

31221 

0.0037801 

99.9953491 

68811 

19881 

0.0024071 

99.9977561 

1391 

6281 

0.0007601 

99.9985171 

5541 

2761 

0.0003341 

99.9988511 

1611 

2471 

0.0002991 

99.9991501 

3891 

1391 

0.0001681 

99.9993181 

QQ  QQCM  701  1 

*88881 

1LJ  \ 

1181 

0.00014^ 

99.9996131 

mm 

99  9997341 

691 

1001 

0.0001211 

99.9998551 

9021 

401 

0.0000481 

99.9999031 

51901 

381 

0 . 000046 1 

99.9999491 

251 

281 

0.0000341 

99.9999831 

1101 

81 

0.0000101 

99.9999931 

59001 

41 

0 . 000005 1 

99.9999981 

33061 

21 

0.0000021100.0000001 

24 


Method  Comparison 


TCPDUMP 


Merge 
PCAP 
files  w/ 

mergecap 


Determine 

PCAP(S) 

that 

contain 

flow 


PCAP  -> 
FLOW 


Write  a 
BPF  filter 
that  will 
return 
session 


1  S£$Sir 


Separate 

Flows 


(q^^j  _  Software  Engineering  Institute  CarnegieMellon 
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Questions? 

CERT  NetSA  tools  website: 
tools.netsa.cert.org 

Contact: 

ecoff@cert.org 

netsa-tools-discuss@cert.org 

netsa-help@cert.org 


£e>  Software  Engineering  Institute  CamegieMellnn 
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Presentation  Abstract 


Finding  a  needle  in  a  PCAP 


It  can  be  difficult  to  find  what  we  are  looking  for  in  a  large  PCAP  repository,  even  when  we  know  what  to 
look  for  and  where  to  look.  When  traffic  captures  start  to  enter  multi-gigabyte  sizes,  the  number  of  tools 
that  can  even  begin  processing  these  files  is  limited.  SiLK  and  other  flow  analysis  tools  provide  the  tools  for 
quickly  narrowing  down  the  search  area  but  when  ground  truth  is  required,  we  are  often  back  to  square  one 
when  searching  for  a  particular  packet  or  flow  in  large  traffic  captures.  This  presentation  will  describe  the 
available  features  in  YAF  for  indexing  large  PCAP  files  with  flow.  We  will  provide  relevant  examples  of 
common  analysis  techniques  with  various  tools  from  the  CERT  NetSA  Security  Suite  and  how  to  perform 
complementary  PCAP  analysis  with  YAF.  This  presentation  will  also  touch  on  deploying  a  tiered  approach 
to  network  monitoring  storage  and  ways  to  maximize  storage  without  compromising  network  analysis. 


(cEKV 
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